Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

Out of curiosity, do you know why the test started failing recently? The hadoop version is the same between 9.0 and 9.1. So I wonder what else has change to cause the failures?

Out of curiosity, do you know why the test started failing recently? The hadoop version is the same between 9.0 and 9.1. So I wonder what else has change to cause the failures?

They all seem to be openjdk21 builds, I wonder if the JDK changed the name of the ForkJoinPool threads?

Does not look like JDK version related to me since the tests do not fail with JDK21 and 8.18.

I've looked back at the JDK source and the ForkJoinPool.commonPool-worker prefix has been like this since at least JDK 17, so I also wonder why tests started failing now -- maybe we tightened the checks? Or maybe we (hadoop) were not leaking any ForkJoinPool thread before?

It looks like for some reason we've moved from using a specific ForkJoinPool instance to using the "common fork join pool"

See that when you create a ForkJoinPool with a specific config, the workerNamePrefix gets set to ForkJoinPool-{pid}-worker-..., which matches the existing pattern.

But the common fork join pool does not have a workerNamePrefix, which causes it to default to ForkJoinPool.commonPool-worker-.... This deviates from the existing pattern.

I tried to reproduce it locally but I can't seem to get leaks from any fork join pool, so it's hard to pin down where hadoop is using it and why that behaviour changed.

Hadoop seems to use both kinds of ForkJoinPool (common and specific) fairly extensively

It would be nice to have an explanation for why this started failing recently, but being the common fork join pool those threads really could be coming from anywhere.

I trimmed down HdfsTests.testSimpleWorkflow() to just creating the repository:

    public void testSimpleWorkflow() {
        Client client = client();
        assertAcked(
            client.admin()
                .cluster()
                .preparePutRepository(TEST_REQUEST_TIMEOUT, TEST_REQUEST_TIMEOUT, "test-repo")
                .setType("hdfs")
                .setSettings(
                    Settings.builder()
                        .put("uri", "hdfs:///")
                        .put("conf.fs.AbstractFileSystem.hdfs.impl", TestingFs.class.getName())
                        .put("path", "foo")
                        .put("chunk_size", randomIntBetween(100, 1000) + "k")
                        .put("compress", randomBoolean())
                )
        );

This fails with the commonPool threads leaked for JDK 22 and 23, but not for JDK 21 and 24 on my machine:

for JDK in 21 22 23 24 ; do ./gradlew ":plugins:repository-hdfs:test" --tests "org.elasticsearch.repositories.hdfs.HdfsTests.testSimpleWorkflow" -Druntime.java=$JDK ; done

Per the ForkJoinPool.commonPool() docs:

This pool is statically constructed; its run state is unaffected by attempts to `shutdown()` or `shutdownNow()`. However this pool and any ongoing processing are automatically terminated upon program `System.exit(int)`.

So it seems reasonable to filter the commonPool threads. But yes, the question remains what the difference is running between these JDKs.

So it seems reasonable to filter the commonPool threads. But yes, the question remains what the difference is running between these JDKs.

I think it's a safe thing to suppress. The use of the common pool could be coming from the JVM libraries. You could stick a breakpoint in the lazy initialisation perhaps? that might reveal where it's coming from. But if it's inconclusive I don't know if it's worth spending a lot of time on?

So it seems reasonable to filter the commonPool threads. But yes, the question remains what the difference is running between these JDKs.

I think it's a safe thing to suppress. The use of the common pool could be coming from the JVM libraries. You could stick a breakpoint in the lazy initialisation perhaps? that might reveal where it's coming from. But if it's inconclusive I don't know if it's worth spending a lot of time on?

The ForkJoinPool.commonPool threads are created from down in Hadoop statistics that use parallelStream(), roughly from running in the debugger with a JDK 21:

HdfsBlobContainer.writeToPath():265-267:

• has a try/finally that closes the FSDataOutputStream,
• that closes the BufferedIOStatisticsOutputStream,
• that closes the RawLocalFileSystem$LocalFSFileOutputStream,
• that has a finally block that cals IOStatisticsAggregator.aggregate()
• which calls IOStatisticsSnapshot.aggregate(), which does several calls to
• IOStatisticsBinding.aggregateMaps() that calls
• EvaluatingStatisticsMap.entrySet() has evalEntries.parallelStream().map(...).collect(...)
  —————— JDK —————
• ReferencePipeline.collect()
• AbstractPipeline.evaluate()
• ReduceOps.evaluateParallel()
• ForkJoinTask.invoke()
• ForkJoinTask.doExec()
• CountedCompleter.exec()
• AbstractTAsk.compute()
• ForkJoinTask.fork()
• DefaultForkJoinWorkerThreadFactory.newCommonWithACC(pool)
• InnocuousForkJoinWorkerThread
• ForkJoinWorkerThread()
• createWorker()

Note that ForkJoinPool.commonPool-worker- is also filtered in TestContainersThreadFilter.

Also it looks like there is a change in DefaultForkJoinWorkerThreadFactory.newThread() between 21 and 24 relating to the security manager. The timing of the test failures lines up with other test failures and updates going on in core infra with security manager and entitlements changes.

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Questions ?

Please refer to the Backport tool documentation

💚 All backports created successfully

Status	Branch	Result
✅	8.18

Questions ?

Please refer to the Backport tool documentation